Claims 

[ C l] i . a system for assessing the performance of an algorithm during development, 

comprising: 

a design of experiments component that establishes an acceptable number of 
experiments for analyzing the algorithm; 

an experiment performance component that runs the established number of 
experiments for the algorithm; and 

a simulation component that simulates the behavior of the algorithm using 
results from the experiment performance component. 

[c2] 2. The system according to claim 1 , wherein the design of experiments 

component selects an appropriate level of experiments for analyzing the 
0 algorithm. 

CI [c3] 3. The system according to claim 2, wherein the design of experiments 

£ component selects the appropriate level of experiments from at least one of full 

J 1 factorial experiments, fractional factorial experiments or screening 

s experiments. 

JjJ [ C 4] 4. The system according to claim 1 , wherein the experiment performance 

fj component comprises a performance metric component that evaluates the 

results of the experiments run for the algorithm. 

[c5] 5. The system according to claim 4, wherein the experiment performance 

component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

[c6] 6. The system according to claim 5, wherein the experiment performance 

component runs the established number of experiments for the algorithm after 
adjustment of logic or parameters. 

[ C 7] 7. The system according to claim 1 , further comprising a module insertion 

component that inserts a module into the algorithm. 



[c8] 



8. The system according to claim 7, wherein the experiment performance 
component runs the established number of experiments for the algorithm 
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including the inserted module. 



W 



[c9] 9. The system according to claim 1 , wherein the simulation component 

performs a Monte Carlo simulation. 

[cl 0] 1 0. The system according to claim 1 , wherein the simulation component uses at 

least one confusion matrix to simulate the behavior of the algorithm. 

[cl 1] 11. The system according to claim 1 , further comprising a simulation 

performance component that evaluates the performance of the simulation of the 
algorithm. 

[cl 2] 1 2. The system according to claim 1 1 , wherein the simulation performance 

component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

[cl 3] 1 3. The system according to claim 1 2, wherein the simulation component runs 

the simulation for the algorithm after adjustment of logic or parameters. 



% [cl 4] 1 4. A system for assessing the performance of an algorithm during 

fT development, comprising: 

Q a design of experiments component that establishes an acceptable number of 

IS experiments for analyzing the algorithm; 

H ! an experiment performance component that runs the established number of 

experiments for algorithm; 

a simulation component that simulates the behavior of the algorithm using 
results from the experiment performance component; and 
a simulation performance component that evaluates the performance of the 
simulation for the algorithm. 

[cl 5] 1 5. The system according to claim 1 4, wherein the design of experiments 

component selects an appropriate level of experiments for analyzing the 
algorithm. 

[ c1 6] 1 6. The system according to claim 1 5, wherein the design of experiments 

component selects the appropriate level of experiments from at least one of full 
factorial experiments, fractional factorial experiments or screening 
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experiments. 

[cl 7] 1 7. The system according to claim 1 4, wherein the experiment performance 

component comprises a performance metric component that evaluates the 
results of the experiments run for the algorithm. 

[cl 8] 1 8. The system according to claim 1 7, wherein the experiment performance 

component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

[cl 9] 1 9. The system according to claim 1 8, wherein the experiment performance 

component runs the established number of experiments for the portion of the 
algorithm after adjustment of parameters. 

[c20] 20. The system according to claim 1 4, further comprising a module insertion 

component that inserts a module into the algorithm. 



pi 

ft f [c21] 21 . The system according to claim 20, wherein the experiment performance 

^ component runs the established number of experiments for the algorithm 

M* including the inserted module. 



[c22] 22. The system according to claim 14, wherein the simulation component 

performs a Monte Carlo simulation. 

[c23] 23. The system according to claim 14, wherein the simulation component uses 

at least one confusion matrix to simulate the behavior of the algorithm. 

[c24] 24. The system according to claim 14, wherein the simulation performance 

component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

[c25] 25. The system according to claim 24, wherein the simulation component runs 

the simulation for the algorithm after adjustment of logic or parameters. 

[c26] 26. A system for assessing the performance of an algorithm during 

development, comprising: 

a design of experiments component that establishes an acceptable number of 
experiments for analyzing the algorithm; 
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an experiment performance component that runs the established number of 
experiments for the algorithm and uses a performance metric to evaluate the 
results of the experiments; 

a Monte Carlo simulation component that simulates the behavior of the 
algorithm using results from the experiment performance component with a 
Monte Carlo simulation; and 

a simulation performance component that evaluates the performance of the 
Monte Carlo simulation for the algorithm. 

27. The system according to claim 26, wherein the design of experiments 
component selects an appropriate level of experiments from at least one of full 
factorial experiments, fractional factorial experiments or screening 
experiments. 

28. The system according to claim 26, wherein the experiment performance 
component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

29. The system according to claim 28, wherein the experiment performance 
component runs the established number of experiments for the algorithm after 
adjustment of logic or parameters. 

30. The system according to claim 26, wherein the Monte Carlo simulation 
component uses at least one confusion matrix to simulate the behavior of the 
algorithm. 

31 . The system according to claim 26, wherein the simulation performance 
component comprises an algorithm adjustment component that adjusts logic or 
parameters of the algorithm for unacceptable results. 

32. The system according to claim 31 , wherein the simulation component runs 
the simulation for the algorithm after adjustment of parameters. 



33. A system for assessing the performance of an algorithm during 
development, comprising: 

a design of experiments component that establishes an acceptable number of 
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experiments for analyzing the algorithm; 

an experiment performance component that runs the established number of 
experiments for the algorithm; 

a performance metric component that evaluates the results of the experiments 
run for the algorithm; and 

an algorithm adjustment component that adjusts logic or parameters of the 
algorithm for unacceptable results. 

[c34] 34. The system according to claim 33, wherein the design of experiments 

component selects an appropriate level of experiments for analyzing the 
algorithm from at least one of full factorial experiments, fractional factorial 
experiments or screening experiments. 

[c35] 35. The system according to claim 33, wherein the experiment performance 

component runs the established number of experiments for the algorithm after 
adjustment of logic or parameters. 

[c36] 36. The system according to claim 33, further comprising a module insertion 

component that inserts a module into the algorithm. 

[c37] 37. The system according to claim 36, wherein the experiment performance 

component runs the established number of experiments for the algorithm 
including the inserted module. 

[c38] 38. A system for assessing the performance of an algorithm during 

development, comprising: 

a Monte Carlo simulation component that simulates the behavior of the 
algorithm with a Monte Carlo simulation, wherein the Monte Carlo simulation 
component uses at least one confusion matrix to simulate the behavior of the 
algorithm; 

a simulation performance component that evaluates the performance of the 
Monte Carlo simulation for the algorithm; and 

an algorithm adjustment component that adjusts logic or parameters of the 
algorithm for unacceptable results. 

[c39] 39 S y S tem according to claim 38, wherein the Monte Carlo simulation 
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component runs the simulation for the algorithm after adjustment of logic or 
parameters. 

[c40] 40. A method for assessing the performance of an algorithm during 

development, comprising: 

using design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm; and 
simulating the behavior of the algorithm using results from the experiments. 

[c41 ] 41 . The method according to claim 40, wherein the using of design of 

experiments comprises selecting an appropriate level of experiments for 
analyzing the algorithm. 

01 [ C 42] 42. The method according to claim 41 , wherein the appropriate level of 

experiments comprise at least one of full factorial experiments, fractional 
factorial experiments or screening experiments. 



nil 



;f ! [ C 43] 43. The method according to claim 40, further comprising using a performance 

P metric to evaluate the results of the experiments run for the algorithm. 

m [c44] 44. The method according to claim 43, further comprising adjusting logic or 

O parameters of the algorithm for unacceptable results. 

[c45] 45. The method according to claim 44, further comprising running the 

established number of experiments for the algorithm after adjustment of logic 
or parameters. 

[c46] 46. The method according to claim 40, further comprising inserting a module 

into the algorithm. 

[c47] 47. The method according to claim 46, further comprising running the 

established number of experiments for the algorithm including the inserted 
module. 

[c48] 48. The method according to claim 40, wherein the simulating comprises using 

at least one confusion matrix to simulate the behavior of the algorithm. 
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[c49] 49. The method according to claim 40, further comprising evaluating the 

performance of the simulation for the algorithm. 

[c50] 50. The method according to claim 49, further comprising adjusting logic or 

parameters of the algorithm for unacceptable results. 

[c5l] 51 . The method according to claim 50, further comprising running the 

simulation for the algorithm after adjustment of logic or parameters. 

[c52] 52. A method for assessing the performance of an algorithm during 

development, comprising: 

using design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
simulating the behavior of the algorithm using results from the experiments; 
and 

evaluating the performance of the simulation for the algorithm. 

[c53] 53. The method according to claim 52, wherein the using of design of 

experiments comprises selecting an appropriate level of experiments for 
analyzing the algorithm from at least one of full factorial experiments, fractional 
factorial experiments or screening experiments. 

[c54] 54. The method according to claim 52, further comprising using a performance 

metric to evaluate the results of the experiments run for the algorithm. 

[c55] 55. The method according to claim 54, further comprising adjusting logic or 

parameters of the algorithm for unacceptable results. 

[c56] 56. The method according to claim 55, further comprising running the 

established number of experiments for the algorithm after adjustment of logic 
or parameters. 

[c57] 57. The method according to claim 52, further comprising inserting a module 

into the algorithm. 



[c58] 



58. The method according to claim 57, further comprising running the 
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established number of experiments for the algorithm including the inserted 
module. 

[c59] 59. The method according to claim 52, wherein the simulation is a Monte Carlo 

simulation. 

[c60] 60, The method according to claim 52, wherein the simulating uses at least one 

confusion matrix to simulate the behavior of the algorithm. 

[c61] 61 . The method according to claim 52, further comprising adjusting logic or 

parameters of the algorithm for unacceptable results. 

[c62] 62. The method according to claim 61 , further comprising running the 

p simulation for the algorithm after adjustment of logic or parameters. 

Jjj; [c63] 63. A method for assessing the performance of an algorithm during 

yj development, comprising: 

using a design of experiments to establish an acceptable number of 
experiments for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
using a performance metric to evaluate the results of the experiments; 
using a Monte Carlo simulation to simulate the behavior of the algorithm using 
results from the performance metric; and 

evaluating the performance of the Monte Carlo simulation for the algorithm. 

[c64] 64. The method according to claim 63, wherein using the design of experiments 

comprises selecting an appropriate level of experiments from at least one of full 
factorial experiments, fractional factorial experiments or screening 
experiments. 

[c65J 65. The method according to claim 63, further comprising adjusting logic or 

parameters of the algorithm for unacceptable results. 

[c66] 66. The method according to claim 65, further comprising running the 

established number of experiments for the algorithm after adjustment of logic 
or parameters. 



■i y 



3 , «. 
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[c67] 67. The method according to claim 63, wherein the simulating comprises using 

at least one confusion matrix to simulate the behavior of the algorithm. 

[c68] 68. The method according to claim 63, further comprising adjusting logic or 

parameters of the algorithm for unacceptable results. 

[c69] 69. The method according to claim 68, further comprising running the 

simulation for the algorithm after adjustment of logic or parameters. 

[c70] 70. A method for assessing the performance of an algorithm during 

development, comprising: 

using a design of experiments to establish an acceptable number of 
experiments for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
evaluating the results of the experiments run for the algorithm with a 
performance metric; and 

adjusting logic or parameters of the algorithm for unacceptable results. 

[c71] 71 . The method according to claim 70, wherein the using of design of 

experiments comprises selecting an appropriate level of experiments for 
analyzing the algorithm from at least one of full factorial experiments, fractional 
factorial experiments or screening experiments. 

[c72] 72. The method according to claim 70, further comprising running the 

established number of experiments for the algorithm after adjustment of 
parameters. 

[c73] 73. The method according to claim 70, further comprising inserting a module 

into the algorithm. 

[c74] 74. The method according to claim 73, further comprising running the 

established number of experiments for the algorithm including the inserted 
module. 

[c75] 75. a method for assessing the performance of an algorithm during 

development, comprising: 

simulating the behavior of the algorithm with a Monte Carlo simulation, wherein 
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the Monte Carlo simulation uses at least one confusion matrix to simulate the 
behavior of the algorithm; 

evaluating the performance of the Monte Carlo simulation for the algorithm; and 
adjusting logic or parameters of the algorithm for unacceptable results. 

[c76] 76. The method according to claim 75, further comprising running the Monte 

Carlo simulation for the algorithm after adjustment of logic or parameters. 

[c77] 77. A computer-readable medium storing computer instructions for instructing 

a computer system to assess the performance of an algorithm during 
development, the computer instructions comprising: 

using design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm; and 
simulating the behavior of the algorithm using results from the experiments. 

[c78] 78. The computer-readable medium according to claim 77, wherein the using of 

design of experiments comprises instructions for selecting an appropriate level 
of experiments for analyzing the algorithm. 



m [c79] 79. The computer-readable medium according to claim 78, wherein the 

IB appropriate level of experiments comprise at least one of full factorial 

y. . experiments, fractional factorial experiments or screening experiments. 

[c80] 80. The computer-readable medium according to claim 77, further comprising 

instructions for using a performance metric to evaluate the results of the 
experiments run for the algorithm. 

[c81] 81 . The computer-readable medium according to claim 80, further comprising 

instructions for adjusting logic or parameters of the algorithm for unacceptable 
results. 

[c82] 82. The computer-readable medium according to claim 81 , further comprising 

instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 

[c83] 83, The computer-readable medium according to claim 77, further comprising 
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instructions for inserting a module into the algorithm. 

[c84] 84. The computer-readable medium according to claim 83, further comprising 

instructions for running the established number of experiments for the 
algorithm including the inserted module. 

[c85] 85. The computer-readable medium according to claim 77, wherein the 

simulating comprises instructions for using at least one confusion matrix to 
simulate the behavior of the algorithm. 

[c86] 86. The computer-readable medium according to claim 77, further comprising 

instructions for evaluating the performance of the simulation for the algorithm. 

[c87] 87. The computer-readable medium according to claim 86, further comprising 

instructions for adjusting logic or parameters of the algorithm for unacceptable 
results. 

[c88] 88. The computer-readable medium according to claim 87, further comprising 

instructions for running the simulation for the algorithm after adjustment of 
logic or parameters. 

[c89] 89. A computer-readable medium storing computer instructions for instructing 

a computer system to assess the performance of an algorithm during 
development, the computer instructions comprising: 

using design of experiments to establish an acceptable number of experiments 
for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
simulating the behavior of the algorithm using results from the experiments; 
and 

evaluating the performance of the simulation for the algorithm. 

[c90] 90. The computer-readable medium according to claim 89, wherein the using of 

design of experiments comprises instructions for selecting an appropriate level 
of experiments for analyzing the algorithm from at least one of full factorial 
experiments, fractional factorial experiments or screening experiments. 

[c91] 91 . The computer-readable medium according to claim 89, further comprising 
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instructions for using a performance metric to evaluate the results of the 
experiments run for the algorithm. 

[c92] 92. The computer-readable medium according to claim 91 , further comprising 

instructions for adjusting logic or parameters of the algorithm for unacceptable 
results. 

[c93] 93. The computer-readable medium according to claim 92, further comprising 

instructions for running the established number of experiments for the 
algorithm after adjustment of logic or parameters. 

[c94] 94. The computer-readable medium according to claim 89, further comprising 

instructions for inserting a module into the algorithm. 

[c95] 95- The computer-readable medium according to claim 94, further comprising 

instructions for running the established number of experiments for the 
algorithm including the inserted module. 

[c96] 96. The computer-readable medium according to claim 89, wherein the 

simulation is a Monte Carlo simulation. 

[c97] 97. The computer-readable medium according to claim 89, wherein the 

simulating uses at least one confusion matrix to simulate the behavior of the 
algorithm. 

[c98] 98. The computer-readable medium according to claim 89, further comprising 

instructions for adjusting logic or parameters of the algorithm for unacceptable 
results. 

[c99] 99. The computer-readable medium according to claim 98, further comprising 

instructions for running the simulation for the algorithm after adjustment of 
logic or parameters. 



[cl 00] 



100. A computer-readable medium storing computer instructions for 
instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 
using a design of experiments to establish an acceptable number of 
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experiments for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
using a performance metric to evaluate the results of the experiments; 
using a Monte Carlo simulation to simulate the behavior of the algorithm using 
results from the performance metric; and 

evaluating the performance of the Monte Carlo simulation for the he algorithm. 

[clOl] 101. The computer-readable medium according to claim 100, wherein using the 

design of experiments comprises instructions for selecting an appropriate level 
of experiments from at least one of full factorial experiments, fractional 
factorial experiments or screening experiments. 

[cl 02] 1 02. The computer-readable medium according to claim 1 00, further 

comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results. 

[cl 03] 1 03. The computer-readable medium according to claim 1 02, further 

comprising instructions for running the established number of experiments for 
the algorithm after adjustment of logic or parameters. 

[cl 04] 1 04. The computer-readable medium according to claim 1 00, wherein the 

simulating comprises instructions for using at least one confusion matrix to 
simulate the behavior of the algorithm. 

[c!05] 105. The computer-readable medium according to claim 100, further 

comprising instructions for adjusting logic or parameters of the algorithm for 
unacceptable results. 

[cl 06] 1 06. The computer-readable medium according to claim 1 05, further 

comprising instructions for running the simulation for the algorithm after 
adjustment of logic or parameters. 



[cl07] 



107. A computer-readable medium storing computer instructions for 
instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 
using a design of experiments to establish an acceptable number of 
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experiments for analyzing the algorithm; 

running the established number of experiments for the algorithm; 
evaluating the results of the experiments run for the algorithm with a 
performance metric; and 

adjusting logic or parameters of the algorithm for unacceptable results. 

[cl 08] 1 08. The computer-readable medium according to claim 1 07, wherein the using 

of design of experiments comprises instructions for selecting an appropriate 
level of experiments for analyzing the algorithm from at least one of full 
factorial experiments, fractional factorial experiments or screening 
experiments. 

[cl 09] 1 09. The computer-readable medium according to claim 1 07, further 

comprising instructions for running the established number of experiments for 
the algorithm after adjustment of logic or parameters. 

[cl 10] 110. The computer-readable medium according to claim 1 07, further 

comprising instructions for inserting a module into the algorithm. 

[cl 1 1] 111. The computer-readable medium according to claim 1 10, further 

comprising instructions for running the established number of experiments for 
the algorithm including the inserted module. 



^ [cl 12] 1 1 2. A computer-readable medium storing computer instructions for 

instructing a computer system to assess the performance of an algorithm 
during development, the computer instructions comprising: 
simulating the behavior of the algorithm with a Monte Carlo simulation, wherein 
the Monte Carlo simulation uses at least one confusion matrix to simulate the 
behavior of the algorithm; 

evaluating the performance of the Monte Carlo simulation for the algorithm; and 
adjusting logic or parameters of the algorithm for unacceptable results. 

[cl 13] 113. The computer-readable medium according to claim 1 1 2, further 

comprising instructions for running the Monte Carlo simulation for the 
algorithm after adjustment of logic or parameters. 
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